What is this document?

We hosted a deepdive on user research with Dharmesh Ba and Abhinav Krishna, where we covered the nuances of how to conduct effective user research, how do you align stakeholders and saw some real life examples of research being conducted.

This document is a summary of the conversation.

Before we begin, a little about our guests:

Dharmesh Ba

Dharmesh is the founder of 1990 Research Labs, where he helps startups like Cred, Jar, Plum with their user research. He comes with a 12+ years of experience being a part of design & research teams at ClearTax, Rupeek and Setu. Apart from this, he runs an amazing newsletter called The India Notes. It's full of insights on the Indian consumer.

Abhinav Krishna

Abhinav is an OG GrowthX member and an OIR. He has 11+ years of experience of working in design & product teams at ClearTax, AgriFi and Microsoft. He’s currently the head of design at RazorpayX and Razorpay Capital.

We covered 2 main topics in the session:

1. How do you go about conducting effective user research?
2. How do you convert your findings into insights?

In this document, we're going the cover the questions asked, and a summary of answers given by panelists.

How do you go about forming research objectives?

Dharmesh says: Do your homework

Before going into any research, conduct your primary and secondary research. You should never go with a clean slate. Always do your homework, research and form some hypothesis on why something might be happening. You should:

• Speak to you team members who are in constant touch with the customers, like the sales staff, customer facing teams etc. They might not be the best people to tell you what to build, but they will tell you why this needs to be built
• Start your conversation who has a perspective on customers
• Study analytics data from google analytics, mixpanel etc. For the product that you're working on, be right on the very basic stuff so that you can build a hypothesis on top of it. For eg. for Razorpay payment gateway, try to figure out the types of customer, AOV, location etc. Then try and understand where the problem lies, is it onboarding, retention, personalisation etc.

For the topic / problem at hand, try and get to the core of it by just data and speaking to customer first teams. Keep asking whys. When you reach a point where you're not able to figure out further, that's when you start research.

But how do you know you can't figure it out further? When you start getting multiple perspectives. That's when you know, you need more truth.

How do you ensure you are not broad enough while starting your research?

Abhinav says: Never take a research objective based document to a meeting

Always take a business problem to a meeting and understand how many business leaders are vested in the problem.

For eg. swiggy wants to increase revenue in swiggy one. Get everyone who's OKRs are related on this in the room. Now you might have a research objective in your mind, there are 2 ways to go about it:

1. Either make everyone think they're getting there on their own
2. Take their advice and modify the initial research objective

This needs to supported by a lot of secondary data, like mentioned above.

There are 3 kinds of research questions:

1. Evaluative: eg. why isn't this screen/filter performing well? Very narrow focused question. Involves both qualitative and quantitative methods and takes like a week to answer
2. Generative: Eg. You are solving for a banking problem, and you want to understand what are the banking services that a typical user expects? A little open ended in nature and takes like 3-4 weeks to figure
3. Foundational: eg. What does management on finances mean to businesses? Usually takes around 2-3 months to form an opinion to conduct a generative research

Given this, to understand the broadness of research objectives, you need to ask your stakeholders:

1. What is the severity of the problem?
2. What is the timeline?
3. What are the number of questions you're ready to take?

When to pushback?

Pushback is when the question that “If this succeeds, what changes?” is not clear, or has too many answers.

For eg. an edtech app wanted to understand why English language learners drop-off in the onboarding. They had observed that the retention of any other language learner was higher than the English language. That seemed very counter-intuitive to them, and they said, “If this succeeds, we will be able to bring up the retention rate to other languages”. Now this is a very well defined problem statement.

To keep it in plain words, do not try to do too many things at the same time.

When the objective is clear, how do you start deriving questions from it?

Dharmesh says: by but thinking about categories of questions.

Typically in one hour, you will be able to 40 good questions. You put everything in your mind on an excel sheet. These will be a lot more than 40 questions. What you should focus on is not asking all the questions but rather cover all the categories of questions.

Now based on the last question, “When was the last time you walked into a branch?” you put a weightage to the answer they gave 2 questions ago about : “Why do you think a branch being near your home is important?”

What this means is if someone says that they opened a bank account in a particular bank because there was a branch nearby, but they say they have never visited that branch, there is some discrepancy in the answers. Then you try to understand “How do they usually resolve their queries/conflicts?”

This means every answer they say, you don’t take it at face value.

So the way to go here is: for the questions that are very close to the focus of the research, you:

1. Explore their answers
2. Put weightage on answers
3. Triangulate

Question: How do you go about conducting the interview?

You need to get them to be vulnerable and get you to tell things. How organically can you get the conversation to start flowing, because that gives to the opportunity to identify the topics you want to deep dive into.

How many questions are too many questions?

The no. of questions do not matter in a user research. What matters is getting stories and understanding the truth. Doesn’t matter if it takes 10, 20, 100, 500 questions to get there.

The only constraint here is the time.

There are usually 3 kinds of personas:

1. People who give one word answers
2. People who tell stories but that irrelevant
3. People who tell relevant stories

Always try to get to the third persona. But don’t try too hard. If you can’t get there, leave it and walk away. You are most likely not getting any insights from that conversation

How do you deduce the trust factor in whatever the participant is saying?

Once the participant believes that you’re asking questions for a genuine reason and not being used against them, that’s a starting point

One way to deduce this is by asking good questions

Truth flows

If you ask any number of whys within a truth, if flows because it’s a part of a narrative. If you have to continuously lie, that’s hard. You can catch the lie in a couple of whys.

Once you identify this, you don’t implicitly mention that you’re lying. You understand, move on and not give much weightage to those questions.

How do you go about conducting the user interview?

Rule 1: Don’t look for the answer. Look for an answer. The only difference is, you need to genuinely interested in your life. Empathise with the participants

Your first job is to make the participant feel comfortable. Don’t directly get to the questions. Try to understand the trivial details about their life like their family, are they native to this place or are they immigrants etc.

Dharmesh says, “When they’re in the field, they try to pick a lot from the surroundings. Maybe there’s a picture of a god, it could be a TV etc”

Before every interview, you make an interview guide that you share with the participants

That interview guide contains the context of the interview, why are you taking the interview and some pointers:

1. You will be recording the call
2. If they have any questions, they can stop the interview and ask questions
3. If they are not comfortable answering questions, they don’t need to answer
4. If they want to stop the interview at any point, they can

You start by asking if they have any questions. Same at the end.

Priming is very essential. Specially in the initial 10-15 mins. In the first 15 mins, you need to:

If you’re working in a product company, it’s very essential to establish that “We’re not testing you, we are having you test our product”. This is because when you establish the conversation as an interview, they try to portray like they know your product which might bias the responses.

Second is: establish the context why you are having this conversation and what will it used for. For ex. if you want to know about their financial details, tell them that hey we are not going to extract any exact details, we are just going to do a proportionality where we say “If you have Rs. X, what are you going to spend it on?”

Third is: get them to a state of vulnerability when they start answering questions. Try to pick a conversation in their area of comfort, and find common talking points to start with.

TO not break the comfort, have only 2-3 people to speak to a participant

Patience is of the utmost importance during your interview

How to research in tier 2 & 3?

There is a lot of cultural difference when it comes to folks from tier 2 and tier 3 towns who let’s say are a maid, autodrivers etc. These people are very transactional, you pay them they get your job done and move on. They don’t really need to respect you as a higher human being.

There are a lot of cultural and societal cues that can differentiate you from someone who’s in these towns. It could be the way you speak, talk, dress, behave etc. You need to convey the fact that you’re the same. They think of you as a big deal because you’ve come from the city, and then they’re continuously trying to guess what you want and give you that.

You need to be cognisant about what you wear, how you speak and behave in order to make them feel comfortable.

How do you know your interview has gone well?

As a general rule of thumb, interviewer should speak for only 20% of the time.

You should get a lot of descriptive stories. Descriptive stories give you layer of informations and hence you get different level of insights. Let’s understand this using an example: If I share my amazon pay account history, how do you figure out my motivation to use it?

1. Observatory: As of today, the participant has a lot of cashback bills, they pay using amazon pay but don’t do any other transactions
2. Behavioural: The participant uses amazon pay only at the start of the month
3. Attitudinal: Because they get some cashboack at the start of every month that they want to extinguish
4. Motivational: Entire usage is pivoted around credit card. CC points are recognised in amazon pay, which pays my annual trip to Europe

You should get a lot of counter-intuitive insights in the first 15-20 mins. That is counter-intuitive because you don’t have context about that point. That means your interview is going right

How should you be collating research findings?

90-95% insights come after the interview, when you sit down and start going through the interviews again.

For Dharmesh, because he comes form a consultation background, he follows a purist form of analysis. Most of the conversation are not in english, so that process here is:

1. Transcribe the call
2. Code/highlight the important parts of the conversation and remove anything unnecessary
3. Put everything on an excel sheet
4. Start tagging. Most of the times, this tagging is the categories that we discussed above
5. You start understanding patterns by looking for answers under a particular category

Every user interview has 2 conversations:

1. The lingual conversation
2. Body language

Along with the lingual conversation, you also need to focus on body language. Understand how a point lands on you and with what intensity has the participant said that point. That’ll help you understand what really is important for the participant.

How do you get from findings to insights?

There are 4 types of insights:

1. Where is the user behaving as expected?
2. Where is the user not behaving as expected?
3. Where is the user emotionally attached in this product?
4. Where is the user chasing a different outcome that you expected them to?

You need to get every stakeholder in a room, sit and go through all videos together and see how they felt about the above 4 questions. Then you prioritise the insights to chase after.

Apart from this, in every research call you’ll find only 40% relevant insights. The rest 60% is not that relevant. Those 60% insights need to go in a separate database, because it might be a visionary insight.

Dharmesh says: At the start of the conversation, we spoke about how research should be conducted when you don’t have answer to certain whys. After the research, you should be able to get that answer and the answer to 3-4 more whys. You need to convince yourself that you are getting closer to the truth. You should be able to convince other stakeholders as well. This will come not just from one conversation, but from a pattern across all conversations.